COMPUTATIONAL NEUROSCIENCE 



REVIEW ARTICLE 

published: 06 December 2013 
doi: 10.3389/fncom.2013.00174 




Exploring the cognitive and motor functions of the basal 
ganglia: an integrative review of computational cognitive 
neuroscience models 

Sebastien Helie 1 *, Srinivasa Chakravarthy 2 and Ahmed A. Moustafa 3 

' Department of Psychological Sciences, Purdue University, West Lafayette, IN, USA 

2 Department of Biotechnology, Indian Institute of Technology, Madras, India 

3 Marcs Institute for Brain and Behaviour and School of Social Sciences and Psychology, University of Western Sydney, Sydney, NSW, Australia 



Edited by: 

Izhar Bar-Gad, Bar-Han University, Israel 
Reviewed by: 

Thomas Boraud, Universite de 

Bordeaux, CNRS, France 

Arvind Kumar, University of Freiburg, 

Germany 

Correspondence: 

Sebastien Helie, Department of 
Psychological Sciences, Purdue 
University, 703 Third Street, West 
Lafayette, IN 47907 USA 
e-mail: shelie@purdue.edu 



Many computational models of the basal ganglia (BG) have been proposed over the 
past twenty-five years. While computational neuroscience models have focused on 
closely matching the neurobiology of the BG, computational cognitive neuroscience 
(CCN) models have focused on how the BG can be used to implement cognitive 
and motor functions. This review article focuses on CCN models of the BG and how 
they use the neuroanatomy of the BG to account for cognitive and motor functions 
such as categorization, instrumental conditioning, probabilistic learning, working memory, 
sequence learning, automaticity, reaching, handwriting, and eye saccades. A total of 
19 BG models accounting for one or more of these functions are reviewed and compared. 
The review concludes with a discussion of the limitations of existing CCN models of the 
BG and prescriptions for future modeling, including the need for computational models of 
the BG that can simultaneously account for cognitive and motor functions, and the need 
for a more complete specification of the role of the BG in behavioral functions. 
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INTRODUCTION 

The basal ganglia (BG) are a group of nuclei at the base of the 
forebrain that are strongly connected to the cortex. While the 
role of the BG had historically been restricted to motor function, 
a substantive amount of recent research suggests that the BG 
are also involved in a variety of cognitive functions (Steiner and 
Tseng, 2010). Behavioral and neural experiments with human 
and non-human animals have informed our understanding of 
the BG function for over a century, and the past two decades 
have seen an increased use of computational models to simulate 
the anatomy and functionality of the BG. The most anatomically 
detailed computational neuroscience models seldom go as far 
as simulating complex animal behavior (because of complexity 
issues), but simpler (less anatomically detailed) models can be 
used to simultaneously account for some anatomical details and 
complex animal behavior. The strength of these later computa- 
tional cognitive neuroscience (CCN) models lies in that they can 
simultaneously account for both neuroscience data and behav- 
ioral data (Ashby and Helie, 2011). 

This review article focuses on CCN models of the BG and 
classifies existing models according to cognitive and motor func- 
tion. The remainder of this article is organized as follows. First, 
the anatomy that is usually included in CCN models of the BG 
is reviewed. This anatomy section is incomplete by design, as 
only details that are simulated to account for specific cognitive or 
motor function are included. Next, we review CCN models used 



to simulate six different cognitive functions, namely categoriza- 
tion, instrumental conditioning, probabilistic learning, working 
memory, sequence learning, and automaticity. This presentation 
is followed by CCN models of motor function. Computational 
cognitive neuroscience models of motor functions are separated 
into models of reaching, handwriting, and eye saccades. The 
review concludes with a discussion of the limitations of existing 
CCN models of the BG and prescriptions for future modeling. 
Future directions emphasize the need for CCN models of the 
BG that can simultaneously account for cognitive and motor 
functions, and the need for a more complete specification of the 
role of the BG in the reviewed functions. 

NEUROANATOMY OF THE BASAL GANGLIA 

The BG include the striatum (caudate, putamen, nucleus accum- 
bens), the globus pallidus (GP), the subthalamic nucleus (STN), 
the substantia nigra (SN), the ventral tegmental area, and the 
olfactory tubercle (see Figure 1). The striatum receives the major- 
ity of afferent connections whereas the internal segment of the 
GP Globus pallidus (internal) (GPi) and SN pars reticulate (SNr) 
are the sources of the majority of efferent connections that target 
cortical regions via the thalamus. Based on both structural and 
functional evidence, the striatum is often divided into a ventral 
and a dorsal part. The ventral striatum includes the nucleus 
accumbens, ventromedial portions of the caudate and putamen, 
and the olfactory tubercle. The dorsal striatum, which is usually 
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FIGURE 1 | Functional anatomy of the basal ganglia Note that only 
subdivisions included in most of the reviewed CCN models are 
represented. Purple boxes correspond to areas of the BG while black boxes 
are not included in the BG. Blue arrows represent excitatory (glutamatergic) 
connections while red arrows represent inhibitory (GABA) connections. The 
direct pathway (1) passes through the D1 receptors in the striatum, the 
indirect pathway (2) passes through the D2 receptors in the striatum, and 
the hyperdirect pathway (3) passes through the STN. If the thalamic 
projections target the same cortical region that initially targeted the 
striatum, the circuit is called a closed loop. Otherwise, the circuit is an 
opened loop. 



the main focus of CCN models of the BG, includes the remainder 
of the caudate and putamen. 

Virtually all of the neocortex sends excitatory (glutamatergic) 
projections to the striatum (Reiner, 2010). Corticostriatal input is 
massively convergent with estimates ranging from 5,000 to 10,000 
cortical neurons converging on a single striatal medium spiny 
neuron (MSN; the main striatal projection neurons) (Kincaid 
et al., 1998). Classically, corticostriatal organization is thought to 
follow a fairly strict spatial topography (Kemp and Powell, 1970). 
Along the rostral-to-caudal extent of the BG, the cortical afferents 
tend be more prevalent from rostral-to-caudal cortical regions. 
For instance, ventral striatum receives input predominantly from 
orbitofrontal cortex, ventromedial prefrontal cortex, and ante- 
rior cingulate cortex (ACC). As one moves caudally within the 
striatum, inputs from areas 9, 46, and 8 become more prevalent 
(Haber et al., 2006; Calzavara et al., 2007), followed by inputs 
from premotor regions (area 6) with the most caudal motor and 
somatosensory cortical regions projecting preferentially to the 
caudal putamen (Flaherty and Graybiel, 1994). Spatial topogra- 
phy holds as you continue rostrally and ventrally through parietal 
and temporal cortices as well as other extrastriate visual areas 
(Kemp and Powell, 1970; Yeterian and Pandya, 1993, 1995, 1998). 

The thalamus provides another major source of input to 
the BG (Wilson, 2004), with the majority of thalamostriatal 
projections originating from the parafascicular-centromedian 
(CMPf) complex (Smith et al, 2010). Thalamic input to 
the striatum synapses on both MSNs and cholinergic toni- 
cally active neurons (TANs; a class of large-body interneu- 
rons) (Smith et al., 2004), with the latter likely playing an 



important role in modulating cortico-striatal synaptic plasticity 
(Ashby and Crossley, 2011). Finally, thalamic input to the stria- 
tum is in a position to modulate BG function by virtue of 
cortico-thalamo-striatal connections and striatal-thalamo-striatal 
feedback (Smith et al, 2010). 

The BG also receives dopaminergic input that plays a critical 
role in modulating striatal activity. Dopamine is projected from 
the ventral tegmental area and SN pars compacta to the BG and 
prefrontal cortex, among other brain regions. Dopamine firing 
patterns fluctuate between two different modes: phasic and tonic. 
While the phasic mode is fast-acting and spans milliseconds, the 
tonic mode is long-acting and can span minutes or hours. The dis- 
sociable function of both phasic and tonic dopamine is debatable 
(Dreher and Burnod, 2002; Assadi et al, 2009; Moustafa et al, 
2013). However, various studies suggest that phasic dopamine 
plays a key role in synaptic plasticity and reinforcement learning 
(Wickens et al., 1996; Reynolds et al., 2001), while tonic dopamine 
is important for speeding-up reaction times (Niv et al., 2007; 
Moustafa et al., 2008) and controlling the signal-to-noise ratio 
(Durstewitz and Seamans, 2008). 

Information flow through the BG follows two distinct path- 
ways (see Figure 1). Striatal MSNs in the direct pathway project 
directly to the output nuclei (e.g., GPi) and selectively express Dl- 
like receptors (i.e., Dl and D5; Gerfen et al., 1990). The striatal 
MSNs in the indirect pathway project to the external segment of 
the GP Globus pallidus (external) (GPe) prior to reaching the 
output nuclei of the BG (e.g., GPi), and selectively express D2- 
like receptors (i.e., D2, D3, and D4; Gerfen and Young, 1988). In 
addition to the direct and indirect pathways, the STN is another 
major input structure of the BG receiving extensive cortical and 
thalamic input. This so-called hyperdirect pathway provides a 
means by which frontal cortical regions can monosynapically 
influence STN function (Nambu et al., 2002). 

With abundant dopamine receptors in the BG affecting the 
dynamics of the different pathways, most CCN models of the 
BG include a role for dopamine. One important way of testing 
whether the hypothesized role for dopamine in the model is ade- 
quate is to simulate the model under dopamine-depleted condi- 
tions. Specifically, reducing the amount of dopamine available in 
the model should produce Parkinsonian symptoms. Parkinson's 
disease (PD) is caused by the accelerated death of dopamine 
producing neurons. The pattern of cell loss is opposite to that 
of, and more severe than in, normal aging. Within the SN pars 
compacta, cell loss is predominately found in the ventral tier 
with less (but still extensive) damage in the dorsal tier (Fearnley 
and Lees, 1991; Gibb and Lees, 1991). In contrast, normal aging 
yields substantially less cell loss and in a dorsal-to-ventral pattern. 
Parkinsonian motor symptoms appear after a loss of 60-70% of 
SN pars compacta cells and 70-80% of dopamine levels in striatal 
nuclei (Bernheimer et al., 1973; Gibb and Lees, 1991). Motor 
symptoms include tremor, rigidity, bradykinesia, and akinesia. In 
addition to motor deficits, non-demented PD patients present 
cognitive symptoms that resemble those observed in patients with 
frontal damage. Numerous studies documenting cognitive deficits 
of PD patients have revealed impairment in a variety of tasks 
related to memory, learning, visuospatial skills, and attention 
(e.g., Gotham et al, 1988; Price et al., 2009). 
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COGNITIVE FUNCTION 

While many cognitive functions have been attributed to the BG 
(for a review, see Steiner and Tseng, 2010), relatively few have 
been modeled and numerically simulated using CCN models, i.e., 
models that can simultaneously account for both neurobiologi- 
cal and behavioral data. Hence, this review does not constitute 
an attempt at reviewing all the cognitive and motor functions 
attributed to the BG: the focus is on CCN models of the BG. 
Note that the model descriptions included are conceptual, in that 
implementation details and mathematical formulations are not 
discussed. The reader is referred to the cited original papers for 
model details and equations. Table 1 summarizes the reviewed 
models along with their respective components. 

CATEGORIZATION 

Categorization is the ubiquitous process by which individual 
items are grouped to form categories. The massive convergence 
of cortico-striatal connectivity makes the BG an ideal site for 
categorization, and much research supports the role of the BG in 
category learning (for a review, see Seger, 2008). 

Models 

One of the earliest and most prominent neurobiological models 
of categorization is called COVIS (Ashby et al, 1998). COVIS is a 
multiple-system theory that was originally developed to account 
for the many behavioral dissociations between verbal and non- 
verbal categorization (as described by the general recognition 
theory; Ashby and Gott, 1988). COVIS includes an hypothesis- 
testing system and a procedural learning system. The hypothesis- 
testing system can quickly learn a small set of (e.g., verbal) 
categories (those that can be found by hypothesis-testing and 
often be verbally described) while the procedural learning system 
can learn any type of arbitrary categories in a slow trial-and-error 
manner (e.g., non-verbal). Each categorization system relies on 
a separate brain circuit but, interestingly, they both include the 
BG. In the hypothesis-testing system, the BG is used to support 
working memory maintenance and for rule switching. In the 
procedural system, the BG is used to learn stimulus — response 
associations. The COVIS model of categorization has been used 
to simulate a large number of category learning experiments and 
made several behavioral predictions, many of which have been 
later confirmed by empirical experiments (for a review, Maddox 
and Ashby, 2004). For example, COVIS predicts that delaying 
the feedback in verbal categorization should not affect perfor- 
mance (because the hypothesis-testing system relies on working 
memory) whereas delaying feedback in non-verbal categoriza- 
tion should impair learning (because the procedural learning 
system relies on dopamine-mediated reinforcement learning in 
the BG). This prediction was confirmed in Ashby et al. (2003). 
In addition, reducing dopamine levels in COVIS can account for 
many cognitive symptoms in PD patients such as perseveration, 
reduced sensitivity to negative feedback, and others (see Helie 
et al., 2012a,b). Likewise, dopamine elevation can account for the 
effect of positive affect on verbal category learning (Helie et al., 
2012b). While most COVIS simulations have used a rate version 
of the model, a spiking version of the procedural-learning system 
has been used to account for some categorization results and 
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Note. DP = Direct pathway 1(1) in Figure 11; IP - Indirect pathway 1(2) in 
Figure 11; HP = Hyperdirect pathway 1(3) in Figure 11; Str = Striatum; GPi 



= Globus pallidus (internal); GPe = Globus pallidas (external); STN = Subthalamic 
nucleus. ' These models used the substantia nigra pars reticulate (SNr) as 
their output node of the basal ganglia. In this context, the SNr is functionally 
equivalent to the GPi. 



extended to account for instrumental conditioning (Ashby and 
Crossley, 2011) and automaticity (Ashby et al., 2007). 

As an alternative, Moustafa and Gluck (2011a,b) proposed a 
computational model of the striatum and prefrontal cortex that 
focuses on the dopamine projections to these areas as well as 
their interactions during multi-cue category learning. In this task, 
participants learn to select and pay attention to a single cue among 
a multi-cue pattern, and then make a categorization response. 
Participants learn this task via corrective feedback. In the model, 
the prefrontal cortex is essential for attentional selection while 
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the striatum is used for motor response selection. Similar to 
COVIS, the Moustafa and Gluck (2011a,b) model can account 
for categorization deficits in PD patients by reducing dopamine 
levels in both the BG and prefrontal cortex, which is in agreement 
with empirical results (Kaasinen et al., 2001; Silberstein et al, 
2005). Additionally, the Moustafa and Gluck (2011a,b) model 
can account for some effects of dopaminergic and anticholinergic 
medication. The Moustafa and Gluck (2011a,b) model assumes 
that the administration of anticholinergic medications in PD 
interferes with hippocampal function, which is also in agreement 
with empirical studies (Meco et al., 1984; Pondal et al., 1996; 
Ehrt et al., 2010; Herzallah et al, 2010). In contrast, the current 
version of COVIS has not been used to simulate medication effects 
in PD. 

Synthesis 

The reviewed models of categorization both agree that the BG, 
and its interaction with the prefrontal cortex, are essential for 
category learning. Furthermore, they agree that dopamine levels 
in both the BG and prefrontal cortex are important. While COVIS 
(Ashby et al., 1998) has been used to simulate a wider range of 
categorization tasks, the Moustafa and Gluck (2011a,b) model 
has been used to simulate more details in a smaller subset. For 
example, one limitation of the Moustafa and Gluck (2011a,b) 
model is that it does not simulate complex multi-cue learning 
tasks that involve paying attention to more than one stimulus 
(which can be done using COVIS). However, the Moustafa and 
Gluck (2011a,b) model can simulate the effect of dopaminergic 
medication, whereas COVIS has not been used to simulate medi- 
cation effects. One important difference between the COVIS and 
Moustafa and Gluck (2011a,b) model is that COVIS assigns a 
different role for BG and cortical dopamine, namely error signal 
and signal gain (respectively). In contrast, Moustafa and Gluck 
(2011a,b) assign both of these roles to dopamine in both the BG 
and the prefrontal cortex. In addition, an important limitation of 
both models is that they oversimplify the anatomy of the BG by 
not including the indirect and hyperdirect pathways. Future work 
aimed at increasing the biological accuracy of COVIS and the 
Moustafa and Gluck (2011a,b) models may highlight some addi- 
tional key differences between the modeling approaches and allow 
for selecting the most appropriate BG model of categorization. 

INSTRUMENTAL CONDITIONING 

Instrumental conditioning (also called "operant" conditioning) is 
a process by which animals learn to behave in a way that will max- 
imize reward and minimize punishment. In a typical instrumental 
conditioning experiment, a neutral environment is altered and 
begins generating rewards (acquisition phase). Next, the reward 
is removed from the environment and the environment reverts to 
its neutral state (extinction phase). Extinction is usually followed 
by a reacquisition phase, where the reward is introduced again 
in the same neutral environment. Typically, a new behavior is 
learned during the acquisition phase, and progressively disappears 
during the extinction phase. The behavior reappears during the 
reacquisition phase, usually at a much faster rate than during 
the initial acquisition phase. This phenomenon is called fast 
reacquisition. Much evidence implicates the BG in instrumental 



conditioning (e.g., O'Doherty et al, 2004; Yin et al, 2005), but 
the neurobiology underlying extinction and fast reacquisition is 
poorly understood. 

Models 

Ashby and Crossley (2011) proposed a spiking model of the 
direct pathway of the BG (see Figure 1) inspired by the COVIS 
procedural learning system (Ashby et al., 1998) to account for 
instrumental conditioning. The Ashby and Crossley (201 1) model 
focuses on the TANs, a population of cholinergic interneurons 
in the striatum that is rarely included in CCN models of the 
BG. Pakhotin and Bracci (2007) have shown that TANs play 
an important role in inhibiting cortical activation of the MSNs 
(i.e., the projection neurons generally modeled in the direct and 
indirect pathways). As suggested by their name, TANs have a 
high baseline firing rate, but they learn to pause in rewarding 
contexts (Apicella, 2007). Ashby and Crossley (2011) suggest that 
one possible role for the TANs is to protect striatal learning 
from catastrophic interference and allow for fast reacquisition. In 
addition to the direct pathway, the Ashby and Crossley (2011) 
model includes a sensory association area, the supplementary 
motor area (SMA), and the CMPf complex. 

The Ashby and Crossley (2011) model is an opened loop 
through the BG (from sensory association cortex to the SMA). 
The stimulus activates the sensory association cortex, which in 
turn activates the striatum and the direct pathway of the BG. At 
the same time, the context activates the CMPf complex, which 
transmits activation to the TANs (this pathway is not included 
in Figure 1). At the beginning of an experiment, the simulated 
subject does not know that the context is rewarding. Hence, the 
TANs do not pause, and the MSNs in the direct pathway cannot 
be activated by the sensory association cortex. This prevents 
stimulus — response association learning. During the acquisition 
phase, the TANs quickly learn that the context is rewarding and 
pause. The MSNs are thus released from inhibition and the model 
learns to produce the rewarding behavior using reinforcement 
learning. Next, during the extinction phase, the TANs learn that 
the context is no longer rewarding and stop pausing. This change 
inhibits the MSNs and interrupts cortico-striatal learning. Hence, 
the associations that were learned during the acquisition phase 
are protected. Finally, during the reacquisition phase, the context 
becomes rewarding again, and the TANs learn to pause. The 
MSNs are released from inhibition, and the learned associations 
are in the same state as in the acquisition phase, which pro- 
duces fast reacquisition. Using this process, the model has been 
used to reproduce the acquisition, extinction, and fast reacquisi- 
tion phases typical of instrumental conditioning and single-cell 
recording data from TANs showing that the cells learn to pause in 
rewarding contexts (Ashby and Crossley, 2011). 

Synthesis 

The Ashby and Crossley (2011) model is the only CCN model 
of instrumental conditioning that can simultaneously account 
for behavioral (e.g., fast reacquisition) and single-cell data (from 
the TANs). This model constitutes an important step in that it 
provides an implementation and numerical simulation of the 
theory that TANs learn to pause in rewarding contexts, and how 
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this can affect reinforcement learning in the BG. However, the 
neuroanatomy of the BG was simplified in that only the direct 
pathway through one of the cortico-BG loops was included. It is 
unclear at this time how the TANs' dynamics would affect the 
indirect pathway, or how the model would behave if more than 
one loops was included. Future work is needed to verify how the 
theory implemented in Ashby and Crossley (2011) behaves in a 
more detailed model of the BG. 

PROBABILISTIC LEARNING 

Probabilistic learning typically refers to tasks where the associ- 
ation between the response and the reward is uncertain. Unlike 
most categorization experiments, the same response to the same 
stimulus can result in different outcomes on different trials. While 
probabilistic learning has been shown to rely on the BG since the 
mid-1990s (Knowlton et al, 1996), it took a decade before CCN 
models of the BG were used to attempt to account for probabilistic 
learning. 

Models 

The Frank (2005) model was originally proposed to account for 
cognitive deficits in parkinsonism. The model includes both the 
direct and indirect pathways through the BG (see Figure 1), the 
premotor cortex, and an unspecified input area (probably located 
in posterior cortex) (so the model is an opened loop). In the 
Frank (2005) model, the input activates both the premotor cortex 
and the striatum. However, cortical activation is insufficient to 
produce a response, so BG processing is required to gate the 
correct response. The focus of the model is on: (1) the role of 
the indirect pathway in probabilistic learning and (2) the role of 
dopamine in probabilistic learning. In the Frank (2005) model, 
the direct pathway is in charge of selecting the appropriate action 
(Go) whereas the indirect pathway is in charge of inhibiting 
inappropriate actions (NoGo). The direct and indirect pathways 
converge in the GPi and compete to control GPi activation, and 
eventually the response. Simulation results show that removing 
the indirect pathway in the model reduces performance, sug- 
gesting that both the direct and indirect pathways are essential 
in probabilistic learning. In addition, the effect of the indirect 
pathway needs to be specific to each action (so that the indirect 
pathway can individually inhibit each action). 

As described in the neuroanatomy section above, the com- 
petition between the direct and indirect pathways is modulated 
by dopamine (the second focus of the Frank (2005) model). 
Specifically, higher dopamine levels increase activation in the 
direct pathway (e.g., through Dl receptors) and reduces activation 
in the indirect pathway (e.g., through D2 receptors). Hence, 
dopamine release following unexpected rewards results in long- 
term potentiation (LTP) in the direct pathway and long-term 
depression (LTD) in the indirect pathway. In contrast, dopamine 
dips following the unexpected absence of a reward reduces activa- 
tion and produces LTD in the direct pathway but increases acti- 
vation and produces LTP in the indirect pathway. The simulation 
results suggest that the dynamic range of the dopamine signal is 
crucial in probabilistic learning and reversal learning (e.g., when 
the response — reward associations are changed during learning). 
Reducing (to simulate PD) or increasing (to simulate medication 



overdose) dopamine levels can result in simulated Parkinsonian 
symptoms (Frank, 2005). 

Another interesting model of probabilistic learning was 
recently proposed by Guthrie et al. (2013). The Guthrie et al. 
(2013) model is based on an earlier computational neuroscience 
model of the BG that focuses on the interaction between the direct 
and hyperdirect pathways (Leblois et al., 2006). The Guthrie et al. 
(2013) model includes two cortico-BG closed-loop that interact 
in the striatum. The first loop is called the "cognitive" loop and 
is used to identify the visual symbols used in the probabilistic 
learning task. The second loop is called the "motor" loop and is 
used to select a response based on the observed symbols. Some 
of the corticostriatal projections affect both loops, but the rest 
of the circuit is segregated. In both loops, the direct pathway is 
in charge of selecting the correct channel (i.e., identifying the 
symbols or the response) while the hyperdirect pathway sends 
non-specific inhibition to the GPi to produce a center-surround 
decision process. All corticostriatal synapses are plastic (using 
dopamine-mediated reinforcement learning) and the cognitive 
loop gradually learns to bias the motor loop, thus producing 
faster reaction times. The model successfully reproduces both 
neural firing rates and behavioral data in the double-arm bandit 
task. 

The categorization models reviewed earlier have also been 
applied to probabilistic learning. The Moustafa and Gluck (20 1 la) 
model focused on the role of dopamine in probabilistic learning. 
In addition to simulating probabilistic learning with normal 
dopamine levels, Moustafa and Gluck (2011a) have simulated 
the effect of decreased dopamine (as in PD) and the effect of 
dopaminergic medication in both the BG and prefrontal cortex. 
The COVIS model has also been used to simulate probabilistic 
learning (Helie et al., 2012a). While COVIS was not used to 
simulate medication effects, the model could account for prob- 
abilistic learning with normal and reduced (as in PD) dopamine 
levels (with a dosage effect such that lowest levels of dopamine 
produced worst performance; see Knowlton et al., 1996). 

Synthesis 

The reviewed models of probabilistic learning tend to be more 
biologically detailed than the reviewed models of categorization. 
Specifically, the Frank (2005) model includes the direct and indi- 
rect pathways, whereas the Guthrie et al. (2013) model includes 
the direct and hyperdirect pathways. In contrast, COVIS (Ashby 
et al, 1998) and the Moustafa and Gluck (2011a,b) models only 
included the direct pathway. Interestingly, however, the Frank 
(2005) model does not include the same details as the Guthrie 
et al. (2013) model. Both models include the direct pathway 
for action selection, and use dopamine-mediated reinforcement 
learning to learn corticostriatal synapses. However, the Frank 
(2005) model uses the indirect pathway as a channel-specific exci- 
tatory process to cancel inappropriate actions whereas Guthrie 
et al. (2013) use the hyperdirect pathway as a non-specific exci- 
tatory process to cancel inappropriate actions. Neither model 
includes both the indirect and hyperdirect pathways. While there 
is agreement on the need for an excitatory process to enhance the 
contrast between the selected and non-selected actions, the exact 
process is still to be determined. 
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While the categorization models only included the direct path- 
way of the BG, one of their strengths is that, in addition to their 
generality, they also include other brain areas. For instance, Unlike 
the Frank (2005) model, the Moustafa and Gluck (2011a,b) model 
simulates the role of prefrontal cortex and its dopamine projec- 
tions, which is in agreement with empirical studies (Mulder et al., 
2003; Histed et al., 2009). Also, analysis of the parameter space in 
the COVIS simulations challenges the role of the BG for proce- 
dural learning in probabilistic learning, and suggests instead that 
the BG are used for hypothesis-testing in this task (Gluck et al., 
2002). So, both categorization models agree on an important role 
for the prefrontal cortex in probabilistic learning, and this role is 
missing from both the Frank (2005) and the Guthrie et al. (2013) 
models. The most productive future approach might be to add 
a prefrontal cortex to the existing probabilistic learning models 
and see how this addition affects the dynamic of the different BG 
pathways. 

WORKING MEMORY 

Working memory is a cognitive function used to maintain and 
manipulate information in real-time for a short duration (sec- 
onds). While working memory has traditionally been associated 
with the prefrontal cortex (Fuster, 2008), Monchi et al. (2000) 
proposed that the BG may be required to maintain information 
in prefrontal cortex. 

Models 

The Monchi et al. (2000) model was originally proposed to 
account for working memory deficits in PD and schizophrenia. 
The model includes three BG-thalamocortical closed loops: two 
with the prefrontal cortex (one for spatial information and the 
other for object information), and one through the ACC (for 
strategy selection). In all three cases, only the direct pathway 
through the BG was included (see Figure 1). The role of the two 
prefrontal-BG loops is to maintain working memory information 
about the stimuli, whereas the ACC maintains the adopted strat- 
egy by inhibiting all the prefrontal cortex loops except one (i.e., 
representing the selected strategy). Visual input to the BG comes 
from the posterior parietal cortex (spatial) and inferior temporal 
cortex (object). The model output is located in the premotor 
cortex, and the nucleus accumbens (not shown in Figure 1) is 
used to provide feedback. In the model, the visual stimulus is 
input to the prefrontal cortex loops, and the stimulus activity is 
propagated through the direct pathway of the BG. As a result, the 
thalamus is released from GPi inhibition, and activation produced 
by the stimulus in the prefrontal cortex reverberates through 
closed-loops with the thalamus. When a response is required, 
the prefrontal cortex transfers its activation to the premotor 
cortex. If the response is incorrect, the nucleus accumbens sends 
a feedback signal to the ACC loop, which selects a new strategy 
by switching its inhibition to different prefrontal cortex loops. 
The Monchi et al. (2000) model has been used to simulate 
a delayed response task and the Wisconsin Card Sorting Test. 
Interestingly, reducing the connection strengths within the BG- 
thalamocortical loops produces Parkinsonian symptoms, whereas 
reducing nucleus accumbens activity produces deficits similar to 
those observed in schizophrenia (Monchi et al., 2000). 



Five years later, Ashby et al. (2005) proposed the FROST 
model to account for intact spatial working memory main- 
tenance. Similar to the Monchi et al. (2000) model, FROST 
includes the direct pathway of the BG (see Figure 1), and working 
memory maintenance relies on reverberating activation between 
the prefrontal cortex and the thalamus. However, unlike in the 
Monchi et al. (2000) model, only one prefrontal cortex loop is 
included, and thalamic activation is not sufficient to maintain 
prefrontal activity: a second set of closed-loops between the 
prefrontal cortex and posterior cortex needs to be simultaneously 
activated to maintain working memory information. In Ashby 
et al. (2005), the focus is on simulating spatial working memory 
tasks, and the area of posterior cortex required for working 
memory maintenance is the posterior parietal cortex. However, 
it is likely the case that the specific location in posterior cortex 
depends on what information is being maintained. For instance, 
if the items being maintained in working memory were objects, 
then it is likely that the posterior cortex area involved would 
be inferior temporal cortex. Another difference between FROST 
and the Monchi et al. (2000) model is that the striatum in 
FROST is activated by a different population of prefrontal neu- 
rons (separate from the working memory maintenance prefrontal 
population) whereas the same prefrontal neurons are used to 
activate the striatum and maintain information in Monchi et al. 
(2000). As a result, the striatum becomes activated only after 
the stimulus has disappeared in FROST, whereas the striatum 
becomes activated as soon as the stimulus appears in Monchi 
et al. (2000). These differences between FROST and Monchi 
et al. (2000) were motivated by recent single-cell recording 
results reviewed in Ashby et al. (2005). FROST has been used 
to reproduce single-cell recordings from many experiments in 
several brain regions, in addition to accounting for working 
memory capacity limitation and the relation between memory 
span and the ability to ignore distracting information (Ashby 
et al., 2005). 

One common theme of the two previous models is that 
working memory activity is maintained by closed-loops between 
the thalamus and prefrontal cortex, and the main role of the 
BG is to release the thalamus from inhibition and allow for the 
reverberating activation to take place. However, this view was 
challenged by Frank et al. (200 1 ) who proposed a model of BG- 
prefrontal cortex interaction in working memory. Specifically, 
Frank et al. (2001) argued that in order for the thalamus to 
contribute to working memory maintenance in the way described 
by the previous models, it would need to have a dedicated 
number of cells comparable to the number of cells dedicated 
to working memory in prefrontal cortex (which is unlikely). 
Instead, the Frank et al. (2001) model proposes that working 
memory maintenance is accomplished by reverberating loops 
between two cell populations within the prefrontal cortex. Similar 
to Monchi et al. (2000) and FROST (Ashby et al, 2005), the 
Frank et al. (2001) model includes the direct pathway through 
the BG (see Figure 1), but the role of the direct pathway is to 
"turn on the switch" on the prefrontal cortex cells and allow for 
reverberating activation. The "switch" can only be turned on if 
the prefrontal cortex cells from one population simultaneously 
receive activation from the BG and the other prefrontal cortex 



Frontiers in Computational Neuroscience 



www.frontiersin.org 



December 2013 | Volume 7 | Article 174 | 6 



Helie et al. 



CCN models of BG function 



cell population. Once the switch is "on", the BG is no longer 
required for working memory maintenance. The Frank et al. 
(2001) model has been used to simulate the 1-2-AX task, a 
working memory task that requires maintenance but also switch- 
ing and selecting new items (Frank et al., 2001). Specifically, 
the 1-2-AX task requires the subject to maintain two cues in 
working memory in order to correctly select a response to a 
target sequence. The identity of the target sequence depends 
on the previous cue, which is used to trigger selection and 
switching. 

One topic that was not addressed by any of the previ- 
ous models of working memory is learning. How can the 
brain learn what is important, and what needs to be main- 
tained in working memory? Moustafa and Maida (2007) 
proposed a computational model of prefrontal cortex and 
BG interactions that is similar to the Frank et al. (2001) 
model except that Moustafa and Maida (2007) also simulate: 
(a) temporal difference learning based on phasic dopamine sig- 
naling and (b) more than one corticostriatal loops that are 
responsible for both motor and cognitive processes. Specif- 
ically, the model includes a cortico-striatal motor loop and 
a cortico-striatal cognitive loop whose functions are action 
selection (choosing motor responses) and cognitive selec- 
tion (choosing the perceptual information to be maintained 
in working memory), respectively. The model can account 
for the separate roles of the motor and cognitive loops 
in working memory maintenance, including delayed-response 
tasks. 

Schroll et al. (2012) recently proposed a CCN model of 
working memory to address the problem of learning complex 
working memory tasks. The Schroll et al. (2012) model includes 
two separate BG-thalamocortical closed loops, one through the 
prefrontal cortex for maintenance and another through motor 
cortex to produce a response. Only the direct pathways were 
used for maintenance and response selection, but the hyper- 
direct pathway was also included in the prefrontal loop as a 
reset mechanism (see Figure 1). Specifically, visual information 
enters the model through the inferior temporal cortex, which 
then activates the lateral prefrontal cortex. This activation is 
transferred through the direct pathway of the BG and releases 
the thalamus from inhibition, which in turn activates the lateral 
prefrontal cortex. In the Schroll et al. (2012) model, working 
memory activation in the prefrontal cortex is maintained by 
a reverberating activation loop through the direct pathway, so 
the BG does not only act as a gating mechanism but is part 
of the maintenance loop (unlike Monchi et al., 2000; Frank 
et al., 2001; Ashby et al., 2005). At any moment, the prefrontal 
cortex can activate the STN, which increases activation in the 
GPi and interrupt working memory maintenance (i.e., the reset 
mechanism). More importantly, the connectivity between the 
prefrontal cortex and the striatum and the connections between 
the prefrontal cortex and the STN are learned using dopamine- 
mediated reinforcement learning. Hence, the model can automat- 
ically adapt and only maintain information that is rewarded in 
working memory. The model has been used to simulate a delayed 
response task, a delayed alternation task, and the 1-2-AX task 
(Schroll et al., 2012). 



Synthesis 

Working memory is one of the most active areas for CCN mod- 
eling of the BG. Five different models were reviewed, each having 
both commonalities and differences. First, all five models focused 
on the interaction between the BG and the prefrontal cortex, but 
only included the direct pathway of the BG for working memory 
maintenance and response selection. Hence, the neuroanatomy 
of the BG was not very detailed. Also, all models except Schroll 
et al. (2012) used the BG as a gating mechanism that turns 
working memory maintenance "on" or "off". The main difference 
is that Monchi et al. (2000) and Ashby et al. (2005) used the 
BG to gate closed loops between the prefrontal cortex and the 
thalamus, whereas Frank et al. (2001) and Moustafa and Maida 
(2007) used the BG to gate closed loop between two populations 
of prefrontal cortex units. This differs from Schroll et al. (2012) 
where the BG was not used for gating, but instead was part 
of the working memory maintenance mechanism itself (i.e., the 
closed loop went through the BG). In all cases, however, working 
memory maintenance was achieved by closed loop through the 
prefrontal cortex. 

Another important difference between the models is that the 
Ashby et al. (2005) and the Moustafa and Maida (2007) models 
focused on simple maintenance tasks. In contrast, the Monchi 
et al. (2000), Frank et al. (2001), and Schroll et al. (2012) models 
were able to simulate more complex tasks involving hierarchical 
structures and switching. Only the Moustafa and Maida (2007) 
and the Schroll et al. (2012) models include learning mechanisms 
that allowed for selecting the relevant information that needs to 
be maintained in working memory. The other models assumed a 
pre-filtering of the information. 

Interestingly, there seems to be a progression and a building 
up of knowledge related to CCN models of working memory. 
The Schroll et al. (2012) model is the most recent, and also the 
most detailed. It is the only model that can learn and simulate 
complex tasks. However, this model departs from all the others 
in that the BG is not used as a gating mechanism but is part 
of the maintenance mechanism. This departure from previous 
modeling is not extensively discussed in Schroll et al. (2012), and 
it is unclear at this point what motivated this departure. More 
work is needed to determine which of these two roles the BG 
play in working memory, but the overlap in the models, and the 
progression in functionality, suggest a steady progress in CCN 
modeling of working memory. 

SEQUENCE LEARNING 

Almost all our everyday behaviors and cognitive activities can be 
interpreted as a sequence of steps that bring us closer to achieving 
a goal. One key question is how can we learn to chain these 
sequences of substeps? Miyachi et al. (1997, 2002) have gathered 
much data suggesting that the BG is important in learning such 
sequences. 

Models 

Nakahara et al. (2001) formalized Miyachi et al. (1997, 2002) 
findings into a CCN model. According to Nakahara et al. (2001), 
sequences are learned in both visual and motor coordinates. The 
visual sequences are learned by a BG-thalamocortical closed-loop 
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linking the anterior striatum with the dorsolateral prefrontal cor- 
tex while motor sequences are learned by a BG-thalamocortical 
closed-loop linking the posterior striatum with the SMA. Only 
the direct pathway through the BG is included in each loop 
(see Figure 1), and both the visual and motor loops learn every 
sequence in parallel using reinforcement learning. The visual loop 
learns faster than the motor loop, and response coordination 
between the loops is controlled by the pre-SMA. According to 
Nakahara et al. (2001), the visual loop relies on working memory 
and is important for rapid acquisition of sequences. However, the 
motor loop is more reliable and produces movement more rapidly 
after training. As a result, control is gradually transferred from the 
visual loop to the motor loop in the Nakahara et al. (2001) model. 
The Nakahara et al. (2001) model has been used to account for: 
(1) the time course of learning (including single-cell recordings 
and lesion studies); (2) the effect of opposite hand use; (3) the 
effect of sequence reversal; and (4) the change in brain locus from 
early to late sequence production (Nakahara et al., 2001). 

Synthesis 

The Nakahara et al. (200 1 ) model is interesting for several reasons. 
First, it successfully accounts for lesion data, single-cell record- 
ings, and behavioral phenomena. In addition, the transition from 
a visual loop to a motor loop represents an early attempt at 
bridging the gap between cognitive and motor functions of the 
BG. However, a recent study by Desmurget and Turner (2010) 
challenges the generality of the Nakahara et al. (2001) model. 
Specifically, Desmurget and Turner (2010) had monkeys perform 
a sequence of visually-cued joystick movements aimed at moving 
a cursor into a pre-determined part of a computer screen. After 
some training, muscimol was injected into the motor segment of 
the GPi to functionally disconnect the BG from the frontal cortex. 
The results show that the kinematics of the movements were 
impaired, but not sequence knowledge. Desmurget and Turner 
(2010) interpreted these results as suggesting that the BG con- 
tributes to motor execution in automatic sequence production, 
but not to the motor sequencing or the storage of the overlearned 
sequence. This result is problematic for the Nakahara et al. (2001) 
model. 

AUT0MATICITY 

Automaticity results from overtraining in a task until perfor- 
mance requires little attentional resources and becomes highly 
rigid (Helie et al., 2010; Helie and Cousineau, 2011). Many com- 
putational models of automaticity development have assigned a 
role for the BG. 

Models 

First, in the Nakahara et al. (2001) model of sequence learning 
(above), automaticity in sequence production is characterized by 
a gradual transfer from the visual loop (which learns sequences 
in visual coordinates) to the motor loop (which learns sequences 
in motor coordinates). This corresponds well with the results 
of Miyachi et al. (1997, 2002), who showed using single-cell 
recordings that task-sensitive cells in early learning are mostly 
located in the anterior striatum whereas selective cells in late 
sequence production are mostly located in the posterior striatum 



(Miyachi et al., 2002). This specialization of the striatum is 
further supported by inactivation studies where muscimol (a 
GABA agonist) was injected in different parts of the striatum 
in early and late sequence production. Well-learned sequence 
production was selectively disrupted by muscimol injection in 
the middle-posterior putamen while early sequence production 
was selectively disrupted by muscimol injection in the anterior 
caudate and putamen (Miyachi et al., 1997). 

However, a recent study by Desmurget and Turner (2010) 
challenges the generality of the Nakahara et al. (2001) model. 
Specifically, injecting muscimol into the motor segment of the 
GPi to functionally disconnect the BG from the frontal cortex 
affects the kinematics of the movements but not sequence knowl- 
edge. These results suggest that the BG contributes to motor 
execution in automatic sequence production, but not to the 
motor sequencing or the storage of the overlearned sequence. 
Interestingly, the results of Desmurget and Turner (2010) are 
consistent with a neurobiological model of automaticity in per- 
ceptual categorization (SPEED) (Ashby et al., 2007). SPEED uses 
the procedural system of COVIS (Ashby et al., 1998) (i.e., the 
direct pathway of an opened loop between posterior cortex and 
premotor areas) but also includes a Hebbian learning mechanism 
between posterior cortex and premotor areas. The role of the 
BG in SPEED is to learn to produce the correct categorization 
responses early in training to ensure that the correct motor plan in 
the premotor areas is consistently activated shortly after the visual 
representation in associative cortex (using dopamine-mediated 
reinforcement learning). This consistent association between 
associative and premotor cortical activity triggers Hebbian learn- 
ing between associative cortex (stimulus) and the premotor areas 
(response), and the direct cortico-cortical connections eventually 
become strong enough so that the BG is no longer required to 
produce a response. When responding becomes purely cortical, 
the skill is said to be "automatic" [note that this is different 
from Nakahara et al. (2001), in which the posterior striatum is 
still required for automatic sequence production]. SPEED has 
been used to simulate single-cell recordings data in many cat- 
egorization experiments, as well as human reaction times and 
accuracies in categorization (Ashby et al., 2007; Helie and Ashby, 
2009). 

While the Hikosaka et al. (2000) and SPEED models can 
account for how behavior becomes automatic, they cannot 
account for how automatic responses are overridden by goal- 
directed behavior when needed (e.g., when the stimulus — 
response associations change). Chersi et al. (2013) proposed a 
computational model of automaticity in instrumental condi- 
tioning that can account for the change back to goal- directed 
behavior when needed. The Chersi et al. (2013) model includes 
the prefrontal cortex (for goal representation), the motor cor- 
tex (for action representation), the sensory cortex (for stimulus 
representation), the BG (for action selection), and the thalamus 
(to relay information between the BG and the motor cortex). Two 
sets of connections are plastic: ( 1 ) connections from the prefrontal 
cortex to the motor cortex (to learn goal — response associations) 
and (2) connections from the sensory cortex to the striatum (to 
learn stimulus — response associations). According to this model, 
the stimulus activates the sensory cortex, which in turn activates a 
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goal in prefrontal cortex and action representations in the stria- 
tum. For automatic behavior, the striatal activation propagates 
through both the direct and indirect pathways (see Figure 1) of 
the BG and an action is selected by inhibiting all but one action at 
the output level (SNr, but it is functionally equivalent to the GPi 
shown in Figure 1). The action that is not inhibited activates the 
appropriate response in motor cortex (through the thalamus). For 
goal-directed behaviors, the prefrontal activation propagates to 
the appropriate action in motor cortex. When an automatic action 
needs to be overwritten by a goal-directed behavior, the prefrontal 
cortex sends activation to the STN, which hyperpolarizes the SNr 
and prevents the BG from controlling the motor response (Chersi 
et al., 2013). The model has been successfully used to account for 
the development of automaticity in an instrumental conditioning 
task and the reversal of stimulus — response associations after 
automaticity had developed (Chersi et al., 2013). 

Synthesis 

The Nakahara et al. (2001) and the Chersi et al. (2013) models 
both assign the role of producing automatic behavior to the BG. 
However, this "classic" role of the BG in automaticity is difficult 
to reconcile with the Desmurget and Turner (2010) data. As an 
alternative, SPEED (Ashby et al., 2007) also assigns an important 
role to the BG in automaticity, but this role is restricted to train- 
ing automatic cortico-cortical projections that can account for 
automaticity. Simply put, the BG is required to learn automatic 
behaviors, but the BG is no longer required to produce automatic 
behaviors once the cortico-cortical connectivity is sufficiently 
strong. The SPEED model can account for the Desmurget and 
Turner (2010) data, but it includes only the direct pathway of 
one loop through the BG. In contrast, the Nakahara et al. (2001) 
model includes two loops through the BG (only the direct path- 
ways) and the Chersi et al. (2013) model includes both the direct 
and indirect pathways, but only one loop through the BG (similar 
to SPEED). In addition to being the most biologically detailed, 
the Chersi et al. (2013) model is the only reviewed model that can 
override automatic behavior using goal-directed behavior. This is 
an important function that was not accounted for by the previous 
models. However, like the Nakahara et al. (2001) model, the 
Chersi et al. (2013) model cannot account for the Desmurget and 
Turner (2010) data. To summarize, each one of these models was 
designed to account for a different aspect of automatic behavior, 
and successfully accounts for the aspect of automaticity for which 
it was designed. The next step is to explore how each one of these 
candidate models can account for the missing function/data that 
was the focus of the other candidate models. 

MOTOR FUNCTION 

This section describes motor functions that have been attributed 
to the BG and that have been simulated using CCN models. 
Hence, computational models that focus only on modeling 
biological data or motor function (e.g., kinematics) were not 
included. Similar to the section reviewing cognitive functions 
above, the model descriptions are conceptual, in that implemen- 
tation details and formalities are not discussed. The reader is 
referred to the cited original papers for details and equations. 



Table 1 summarizes the reviewed models along with their respec- 
tive components. 

REACHING 

The BG has been implicated in reaching movements for many 
years (for a review, see Bischoff, 1998). Not surprisingly, 
PD patients show unmistakable changes in reaching move- 
ments, which can be used for diagnostic purposes (Brown and 
Jahanshahi, 1996). Simple reaching movements in PD patients 
show longer reaction times and movement times than normal 
controls. This reduced movement speed seen in PD reaching 
is called bradykinesia. From a physiological perspective, a typ- 
ical reaching movement under normal conditions consists of a 
sequence of agonist-antagonist bursts. In contrast, a PD reaching 
movement is generally multi-staged and involves multiple ago- 
nist bursts. Furthermore, PD reaching movements have greater 
variability of hand position for larger movements (Sheridan and 
Flowers, 1990). PD patients also show impairment in sequential 
movements (Weiss et al., 1997). For example, during movements 
aimed at reaching a glass of water, PD patients exhibit an inordi- 
nately long pause between the reaching and retrieval of the glass. 

Models 

Several computational models relating dopamine deficiencies to 
impaired reaching movements have been proposed. One of the 
first models of PD reaching movements is the model of Bischoff 
(1998). Bischoff (1998) model includes the prefrontal cortex (for 
working memory/learning), the SMA (for sequence generation), 
the pre-SMA (for sequence preparation), motor cortex (for move- 
ment parameters), the thalamus (to filter information from the 
BG to cortex), and the BG. The BG model assigns complementary 
roles to the direct and indirect pathways (see Figure 1). According 
to Bischoff (1998), the role of the direct pathway is to inform 
the motor cortex of the movement's next sensory state, while the 
role of the indirect pathway is to inhibit competing movements. 
The function of dopamine is to keep the balance between the 
two pathways, which is impaired in PD. The Bischoff (1998) 
model was used to simulate the reciprocal aiming task, a task 
where subjects are asked to alternatively tap a stylus between two 
targets as quickly as possible. Reducing the dopamine levels in the 
simulation reproduced bradykinesia and the exaggerated pauses 
in sequential movements observed in PD. 

Magdoom et al. (2011) also proposed a model of the role of 
the BG in PD reaching movements. The model is cast in the 
framework of reinforcement learning and focuses on interactions 
between the motor cortex and the BG. The Magdoom et al. 
(2011) model uses the classical interpretations of BG pathways 
according to which the direct pathway facilitates movement, (i.e., 
the "Go" pathway), while the indirect pathway inhibits move- 
ments (i.e., the "NoGo" pathway). Switching between the two 
pathways is thought to be driven by striatal dopamine levels. 
However, Magdoom et al. (20 1 1 ) also deviate from the classical 
"Go'VNoGo" model of the BG by adding an intermediate regime 
called the explore regime. The explore regime is used to control 
the stochasticity of action selection when the gradient is absent 
or too weak to allow for reinforcement learning. The indirect 
pathway is proposed to be the substrate supporting the explore 
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regime. Simulations show that under dopamine-deficient condi- 
tions of PD, the model spends less time in the "Go" regime while 
spending more time in the remaining two regimes. These regime 
changes were used to account for a variety of features of impaired 
reaching movements in PD including movement undershoot (Van 
Gemmert et al., 2003), bradykinesia, and increased path variabil- 
ity (Sheridan and Flowers, 1990). 

Synthesis 

Two models that highlight the role of the BG in reaching were 
reviewed. The Bischoff (1998) model includes, in addition to 
the BG, cortical areas such as prefrontal areas, the SMA and the 
pre-SMA. The model captures bradykinesia and abnormal pauses 
in sequential movements under Parkinsonian conditions. The 
reaching model of Magdoom et al. (2011) also incorporates the 
BG and the motor cortex. However, the Magdoom et al. (2011) 
model is cast in the framework of reinforcement learning, whereas 
there is no learning in the Bischoff (1998) model. Focus on 
learning makes the Magdoom et al. (2011) model consistent with 
the proposed role of BG in error correction (Lawrence, 2000). As 
a result, the Magdoom et al. (2011) model is more general and 
is consistent with the view that a wide range of BG functions 
can be explained within the framework of reinforcement learning 
(Chakravarthy et al., 2010). The compatibility of the Magdoom 
et al. (20 1 1 ) model with other CCN models of BG function may 
facilitate integration to achieve a more complete model of BG 
function. 

HANDWRITING 

Handwriting is a fine motor skill. PD patients often exhibit an 
impaired form of handwriting, known as micrographia, char- 
acterized by reduced letter size, a "kinky" handwriting contour, 
and abnormal fluctuations in velocity and acceleration (Teulings 
and Stelmach, 1991; Van Gemmert et al, 1999; Gangadhar et al, 
2009). As a result, handwriting features like stroke size, peak 
acceleration, and stroke duration have been attributed to the BG 
and used for diagnosis of PD (Phillips et al., 1991; Van Gemmert 
et al., 2003). 

Models 

Although models of PD handwriting are scanty, extensive work 
has been done on modeling handwriting generation. One of 
the earliest insights into modeling handwriting consisted of per- 
forming a Fourier-like resolution of handwriting into oscilla- 
tory components (Hollerbach, 1981). This notion has led to the 
development of oscillatory or spiking neural network models of 
handwriting generation that can be trained to produce single 
characters (Schomaker, 1991; Kalveram, 1998). However, the 
models of Schomaker (1991) and Kalveram (1998) suffered from 
the absence of a plausible procedure for initializing the phases 
of neural oscillators, a difficulty that was solved in an oscillatory 
neural network model of handwriting generation proposed by 
Gangadhar et al. (2007). 

While the above-described models did not explicitly include 
the BG, Gangadhar et al. (2008) combined the Gangadhar et al. 
(2007) handwriting generation model with a model of the BG. 
Similar to handwriting patterns observed in PD patients, the 



Gangadhar et al. (2008) model exhibits micrographia under 
conditions of reduced dopamine. Another significant feature of 
the model is the role of the dynamics of the STN and the 
GPe, which are connected as a loop to produce complex oscil- 
lations. Under pathological conditions, the oscillations of the 
STN and the GPe in the model are highly correlated, resembling 
the correlated neural firing from STN and GPe neurons under 
dopamine-deficient conditions observed in real electrophysiol- 
ogy experiments (Bergman et al., 1994; Brown et al., 2001). 
Under the influence of correlated oscillations of STN and GPe, 
the Gangadhar et al. (2008) model produces handwriting with 
large fluctuations in velocity in addition to diminutive letter 
size. 

As another example, Contreras-Vidal and Stelmach (1995) 
attached a BG model to the VITE-WRITE model (Bullock et al, 
1993) to simulate PD handwriting. The Contreras-Vidal and 
Stelmach (1995) model includes the direct, indirect, and hyperdi- 
rect pathways of the BG (see Figure 1), the SMA, and other motor 
and premotor areas. The role of the SMA is to read-in the next 
motor subprogram from the movement plan, while the role of 
the other motor and premotor areas is to produce the movement 
selected by the SMA. The role of the BG is to modulate the 
dynamics of the formation of movement trajectories (produced 
by VITE-WRITE). Reducing dopamine in the model to simulate 
PD results in reduced letter size, as observed in PD patients. 

Synthesis 

Two models of PD handwriting were reviewed above. The model 
of Contreras-Vidal and Stelmach (1995) combines the VITE- 
WRITE model (Bullock et al, 1993) with a BG model. The essence 
of the model consists of showing that dopamine reduction in PD 
causes an imbalance in the outputs of the direct and indirect 
pathways. Although constructed out of considerably different 
modeling components, the model of Gangadhar et al. (2008) 
also shows an imbalance in the activations of the direct and 
indirect pathways under simulated PD conditions, which causes 
a reduction in letter size. In addition, Gangadhar et al. (2008) 
also account for the oscillations in STN-GPe interaction. Loss of 
complexity in these oscillations under PD conditions were linked 
to higher velocity fluctuations and distorted handwriting contour 
in PD handwriting. To summarize, the indirect pathway appears 
to be critical in accounting for handwriting. 

EYE SACCADES 

Eye saccades are rapid, darting eye movements that shift the fovea 
to points of interest in the visual scene. There is an extensive 
cortical and subcortical network that is responsible for saccade 
generation, and the BG play a key role among the subcortical sub- 
strates for saccade generation (Hikosaka et al., 2000). The influ- 
ence of BG on saccades is propagated via the superior colliculus, 
a midbrain nucleus with a central role in saccade generation (not 
shown in Figure 1). Studies on Parkinsonian monkeys prepared 
by MPTP (a neurotoxin used to destroy dopaminergic neurons) 
infusion have observed prolonged saccades, longer reaction times, 
smaller peak velocities, and smaller amplitudes (Kato et al., 1995). 
Smaller peak velocities and smaller amplitudes in PD saccades 
may be comparable to bradykinesia and hypometria found in 
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PD reaching movements. Similarly, analogous to PD tremor in 
extremities, some PD patients exhibit square-wave jerks in visu- 
ally guided saccades (Rascol et al, 1991). 

Models 

Computational modeling literature that specifically focuses on 
the role of BG in saccade generation is rather limited. Dominey 
and Arbib (1992) proposed a model of the role of the BG in 
sequential saccade generation. Their model includes a num- 
ber of relevant neural substrates such as the superior collicu- 
lus, thalamus, frontal eye fields, and the BG. In the Dominey 
and Arbib (1992) model, the BG is used as an indirect link 
between the frontal eye field and the superior colliculus, and 
its main function is to prevent saccades while a target stimulus 
is foveated. As such, only the direct pathway through the BG 
is modeled. The Dominey and Arbib (1992) model has been 
used to simulate simple saccade data, memory saccade data, 
double saccade data, compensatory saccade data, and lesion 
data. 

Two decades later, Krishnan et al. (2011) proposed a model 
of the role of the BG in saccade generation using the principle 
of reinforcement learning. Similar to their model of PD reaching 
movement (Magdoom et al, 2011), the indirect pathway serves 
as an explorer that drives the saccades toward more rewarding 
targets. The model was able to successfully simulate two forms 
of visual search, namely feature search and conjunction search, 
a sequential saccade task, and a directional saccade task. When 
PD-related changes were incorporated in the model by reducing 
BG output, the model exhibited increased search times (Krishnan 
etal.,2011). 

Synthesis 

Two models of the role of BG in saccade generation were reviewed 
above (Dominey and Arbib, 1992; Krishnan et al., 2011). Both 
models can account for a range of saccade data in normal and 
lesioned/pathological conditions. The anatomical components 
incorporated by the two models are also quite similar. However, 
there are two main distinguishing features between the two mod- 
els. One of these features is anatomical: the Dominey and Arbib 
(1992) model does not include the indirect pathway in the BG, 
whereas the indirect pathway plays a key role in the Krishnan 
et al. (201 1) model. The second feature is functional: the Dominey 
and Arbib (1992) model does not involve learning, while that of 
Krishnan et al. (201 1) model is based on reinforcement learning. 
These key differences make the Krishnan et al. (20 1 1 ) model more 
biologically and functionally detailed. 

GENERAL DISCUSSION 

This article presented a review of CCN models of cognitive 
and motor functions. The 19 reviewed models were organized 
to highlight BG functionality and classified according to six 
cognitive functions (i.e., categorization, instrumental condition- 
ing, probabilistic learning, working memory, sequence learning, 
and automaticity) and three motor functions (i.e., reaching, 
handwriting, and visual saccades). On the one hand, some of 
the reviewed models are standalone models of specific func- 
tions of BG, e.g., the reaching model of Bischoff (1998). On 



the other hand, there are models that are specific instances of 
a more general learning framework applied to BG function. 
COVIS (Ashby et al, 1998, 2007; Ashby and Crossley, 2011), 
and the models of Chakravarthy and colleagues (Krishnan et al, 
2011; Magdoom et al., 2011) belong to the second category. 
For example, both the models of Krishnan et al. (2011) and 
Magdoom et al. (2011) used a nearly identical reinforcement 
learning-based approach to account for the specific motor func- 
tions of reaching and saccade generation. A review article by 
Chakravarthy et al. (2010) proposes that an expanded frame- 
work based on reinforcement learning, adapted to BG anatomy 
and physiology, can be used to explain a wide variety of BG 
functions. Such a proposal needs a more extensive modeling 
and experimental investigation for further confirmation. How- 
ever, interestingly, CCN models accounting for more than one 
function were accounting for more than one cognitive func- 
tion or more than one motor function. None of the reviewed 
CCN models could account for at least one motor and one 
cognitive function simultaneously. This may be a serious limi- 
tation as behavioral experiments are beginning to reveal impor- 
tant interactions between motor and cognitive processes. Below, 
we discuss how cognitive processes might impact motor func- 
tion, and point to novel directions for computational modeling 
studies. 

INTERACTION OF MOTOR AND COGNITIVE PROCESSES 

While none of the models included simultaneously accounted 
cognitive and motor functions, they all had a cognitive and motor 
component. For example, the Ashby and Crossley (2011) model 
made a cognitive decision, but it also included premotor areas 
associated with the response. It just did not include a detailed 
model of the motor response (e.g., how is the left button pressed). 
Likewise, the Gangadhar et al. (2007) has to include a cognitive 
component specifying what character is to be drawn. However, 
the focus is on how the movement is produced. Perhaps the model 
that comes closest to integrating motor and cognitive functions is 
the model of Guthrie et al. (2013). In this model, both a cognitive 
and a motor decision are taken, and the interaction between these 
decisions is accounted for. However, this model does not include a 
detailed simulation of how the movement is produced. Therefore, 
it was only discussed in the context of cognitive function. 

One way to explore how cognitive and motor functions 
interact is to explore disease states. For example, akinesia 
and bradykinesia in PD are arguably associated with BG (and 
corticostriatal circuits) dysfunction, while tremor is perhaps 
associated with cerebellar, thalamic, and STN abnormalities 
(Kassubek et al, 2002; Probst-Cousin et al., 2003; Weinberger 
et al, 2009; Zaidel et al., 2009; Mure et al., 2011). For example, 
Schillaci et al. (2011) found that PD patients with akinesia and 
rigidity as the predominant symptoms have significantly more 
widespread dopamine loss in the striatum than PD patients with 
tremor as the predominant symptom. Because these different 
brain areas (e.g., striatum, cerebellum) are also involved in 
different cognitive functions, it is reasonable to hypothesize that 
different PD motor symptoms may be associated with different 
cognitive impairments. Accordingly, Jankovic et al. (1990) found 
that PD patients with predominant tremor are less cognitively 
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impaired than patients with bradykinesia. Below we explore some 
specific PD motor symptoms and their relation to cognition. 

Akinesia 

Experimental studies have shown that PD patients with severe 
akinesia are generally more cognitively impaired than PD patients 
with predominant tremor (Vakil and Herishanu-Naaman, 1998; 
Poletti et al, 2011, 2012; Poletti and Bonuccelli, 2013). For 
instance, PD patients with severe akinesia and rigidity symptoms 
are more impaired than PD patients with severe resting tremor at 
working memory tasks (Moustafa et al., 2013). Likewise, studies 
have shown that PD patients with tremor are usually less cogni- 
tively impaired than PD patients with akinesia or gait dysfunction 
(Burn et al, 2006; Lyros et al, 2008; Oh et al, 2009; Domellof 
et al, 2011). For example, Vakil and Herishanu-Naaman (1998) 
found that tremor-dominant PD patients are less impaired at 
procedural learning than akinesia-dominant PD patients. 

Most motor models of the BG and corticostriatal circuit 
function have been able to explain the occurrence of akinesia 
and bradykinesia, but not tremor (Obeso et al., 2008). We sug- 
gest that motor performance may rely on cognitive processes in 
two different ways: (a) maintenance of motor plans in working 
memory while performing a sequence of movements, such as 
hand/leg movement, grasping, or reaching (Hayhoe et al., 2002; 
Ohbayashi et al., 2003; Piek et al., 2004; Issen and Knill, 2012); or 
(b) maintenance of goals in working memory while performing 
a motor act, such as maintaining the goal of grasping the cup 
in working memory while moving the hands (Batuev, 1989; 
Mclntyre et al, 1998; Matsumoto et al, 2003). This relationship 
between cognitive and motor processes could explain why some 
cognitive training programs are effective at treating motor dys- 
function in PD patients (Disbrow et al., 2012). Although this is 
speculative, computational models are needed to explicitly study 
the complex relationship between motor and cognitive processes 
in healthy subjects and PD patients. 

Freezing of gait 

Freezing of gait — paroxysmal cessation of motor output — is a 
common symptom in advanced PD (Hoehn and Yahr stage 2+) 
(Giladi et al., 1992). Freezing of gait is debilitating since it often 
leads to falls and, importantly, is not manageable by common 
psychopharmacological medications (Giladi et al., 1992; Matar 
etal., 2013). 

Research shows that perceptual and cognitive factors play a 
role in successful locomotion and the occurrence of freezing 
of gait episodes in PD patients (Lewis and Barker, 2009; Nai- 
smith et al., 2010; Matar et al., 2013). For example, providing 
auditory or visual cues or instructions can often reduce the 
occurrence of freezing behavior in PD patients (Lewis and Barker, 
2009). Other studies found that walking dysfunction in PD is 
related to difficulty in resolving response interference produced 
by distractors (Plotnik et al., 2011; Vandenbossche et al., 2011). 
Locomotive dysfunction in PD is associated with brain volume 
changes (Kostic et al., 2012; Tessitore et al., 2012) and aberrant 
neural activity within the prefrontal cortex (Matsui et al, 2005; 
Shine et al., 2013), suggesting a role for cognitive processes in 
locomotion. 



There are currently no computational models that simulate 
the role of cognitive processes in the occurrence of freezing 
of gait in PD patients. Prior computational models of BG- 
cortex interactions have focused on the simulation of cognitive 
processes (O'Reilly and Frank, 2006), learning, or simple action 
selection in static environments (Gurney et al., 2001) without 
considering how cognitive factors might affect motor actions 
such as locomotion. Future models should simulate how the 
cortex represents multiple inputs (including perceptual and 
cognitive) that feed into the BG, which is important for action 
selection (e.g., move right, left, forward, etc.). Future models 
should also be more dynamical in that they should continuously 
receive and update perceptual input from the environment and 
produce motor output (step right, left, . . .), which then takes the 
model to a new perceptual input, and so forth. 

WHAT IS THE ROLE OF THE BASAL GANGLIA IN COGNITIVE AND 
MOTOR FUNCTION? 

In addition to the current unavailability of CCN models of the 
BG that can simultaneously account for cognitive and motor 
function, another limitation of the current state of BG modeling 
is the absence of consensus on the specific function of the BG in a 
given task. For example, many CCN models of working memory 
assign a role for the BG, but some models use the BG as a gating 
mechanism allowing for thalamo-cortical loops (e.g., Monchi 
et al., 2000; Ashby et al., 2005), while others use the BG as a gating 
mechanism for cortico-cortical loops (e.g., Frank et al., 2001) or 
as the actual maintenance mechanism (Schroll et al, 2012). As 
with many other cognitive and motor functions, CCN models 
are critical in pinpointing the specific function of the BG in the 
cognitive task (e.g., working memory). Computational models 
can be simulated to identify the consequences of different design 
choices, and these predictions need to be tested empirically. 
While models tend to do very well at simulating the function that 
motivated the model, it is unclear at this point how the model can 
handle other (different) functions. One way to select useful BG 
CCN models is to consider generalization capabilities. Towards 
this end, general integrative frameworks are most useful. For 
example, the reinforcement learning approach of Chakravarthy 
and colleagues (Krishnan et al, 2011; Magdoom et al., 2011) 
or the COVIS-based approach of Ashby and colleagues (Ashby 
et al, 1998; Apicella, 2007; Ashby and Crossley, 2011) are useful 
because they have been used to account for functions that were 
outside of the original scope of the model. Other models of 
cognitive and motor functions need to be generalized to account 
for data for which they were not originally designed to build 
biological "cognitive architectures". Frameworks that are already 
general should attempt to bridge the gap between CCN models 
of cognitive function and CCN models of motor function. This 
could be achieved by integrating existing models. For example, 
a detailed CCN model of motor function could be added to 
the COVIS framework. Likewise, a detailed CCN model of 
cognitive function could be added to the reinforcement-learning- 
based approach of Chakravarthy and colleagues. While more 
data will help in eliminating some of the candidate CCN BG 
models, generalization and integration will be required to avoid 
overfitting the model to the available data. 
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